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' * The Joint Disseminaltion Review Panel (JDRP) isVa^federal review 

board which examines educational -products and practices and determines */ 
•*whethei8''tfie'\suljmitting project prqvtded persuasive evidence ^o 

'* pe5$.-\Eff^ctivene|s* criterta state tliat there is evidence of positive 
i^pactr tljat effects^are statistically^significant and educationally im- 

../ poHahti that ra^a^ure^ arid valid^ that the observed -^fects 

can. be" attributeci to t^/inte that the evidence is, credible, 

' "and tfrat the intervejition and its effects can be replicated , in other. 



sites. If JDRP review results in approval, the intervention, is recog- 

ni zed as' having the capabilfti. of producing its claimed. effects'". The 

' intervention ,niay then be;lpt:iobally disseminated. Thus the Panel serves 

' as a. screening mechanism for national dis'^emi nation through a review 

process that substantiates the vali(*fty of "the project's* claims,.;. - 

■ ■ ' . ..■ • . ■ , ■ . ./■ ■■. 

The purpose -of the present study was to examine the freview process 
••» , • • ■ ■ .. . . , . ■ 

based on the identified dimensions of evidence of effectiveness. Three ^ 

research questions were asked: 

1. .What were the characteristics of submittal s r^^iewed 



by JDRP? 

Which independent factor§^ifferenti.ir<ed^6etweeh appr/ved 
and rejected submittal; 



3. 




What set of variables. discri mi na^jed between, ap^^ 
and rejected. submttals? . ,''.-/,/ / a' ^' ' 




Td answer these •questions, an ihstr^mervt^ was^evised that o61 1 ected 
data related to the subijjitting pl^ojeet's de/rbgraphic characte^^ the 
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-hatum of the educational intervention, and evidence of : effectiveness. 

: .• IJeVelopmerit of the instrument was based on documents related to the 

^; ...e^^^ the rfivieW;,pr^ess (Educiti on Division, 

■ Notft 1 i ffati dnal Testi ng S&ry^p^i(&%Q^r<± Corporation , Note 2 ; Netv/ork , 

^ «ote -3; Pyecha '&:Tisheri .Note 4; Ta^lmadge, Note 5) and 'evaluation 

.literature .(Beriistei Hi- Bpbr^^^^ Bbrgatta^xiwS; Bracht & Glass, 

1968; GanipbeTl ,' 1969r C^rapblTl S Stanl^yt^9'63; Xook & .Campbel 1 , 1979) . 

The /sample xoflifs'ted of 8ji<^ojects that submitted their inter- 

y^.ventionsy^^ during 1978 and 1979. Of these 83 

' • '^^ on X)n1y one occasion while .the remaining 13'sub- 

litted tWicftf^lR the case of the project^ that submitted oaly once, 

/"52' subnii'ttals were approved, and 18 were rejected. In the 'case of the 

' >lS 'projects -Ihat submitted twice, the first submittal was initially re-, 

'. ■ ■ ... 

■, je'cted while' the resubmission was approved. Since these projects- appeared 

'to be two distinct grolips, they 'w6re examined as, separate sub-saifipTes. 

; ^ The 1 (Scan y-devel oped instrument was field tested prior to admini- 

^ stf-atioh to^each of the 96 submittals. Using ten randomly selected sub- 



I 



mlttals, three reviewers independently read each submittal and recorded the - 
apprqpri.ate data on the instrument. Rater agreefnept was 81% on items. 

" < . ( '• , » ■ , . ■ ■ ■, 

related to , the reporting of evidence of effectiveness. 

Upon , review of ea-ch of the 96 submittals, infbrma^tion^ 
on the instrument was then collected by the investigator. Since a number 
of the submittals reported data on more than one objective, -a decision was' 
made to assess only those objectives that were cognitive in nature. 
Succeeding items then collected information wmch corresponded to the criteria 
employed by JDRP as they related to. the selected cognitive objective. ' 



. These items elicited the folTovring types of da 



Variable (Operational 
. . . * . DefinitionJ 



1. evalliation'defrigri ^ 
• 

j Z. sampTihg procedures 

, • • \ . . x . . 

3. type of. comparison group* ^ , 

»-. 

4. statisticaV sighificance 
5/ educational importance 
6. -internal validity' 

1. external Validity 

.8. replication . ' • > 

. \ . • ■ -"^ . / \ 

9. type of instrument either, 
standardized or locally- , 
developed . . ' . ■ 

10. "reliability Of the instrument 

li: validity of the instruirent 

.12,,, test administration procedures 

13.' types of reported scores. 

't4t- types of data arialysi s procedures 




riterion 



[impjict 

/credibilit^bf evidence 

evidenc^f impact 

evider^ljffl^ statistical - 
of effects 

' ■ . • . ■ ■ 

evidence that the effects 
are educationally me^n^gful 



evidence that the effects aihe ■ 
attributable to the intervention 

*• ■ ■ ' > 

.evidence of general izability 
' evidence of general izabil»ity 
' interpretabi 1 i ty of measures ' • 

intepretability of measures 
interpretabi lity of measures ■ 
credibility of evidence 
credibility" of evidence 
credibility of, evidence „- 



% 



\o' answer research question 1, frequency inform^i on was tabulated 
on each -of %heN instrument's items. These data were cojlecte'd on^th 
sub'-sampies: those vthat- submitted only once and ^^^t^^^ Wt submitted 
twice. Results, indicated' that each Of the 96 submittals corvErtTiec 
measurable objectives, an evaluation design, and a specified sample size 
and administered at least; one instrument. Inclusion of data related to ^ 
the identified evidence" of effectiveness criteria' varied depending on 
the particular }%em. Tables 1,-2, and'"3 present a suramary of the inform- 
ation that submittals contained on selectad variables related to evidence 
of effectiveness;* ' v . . 
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■ i ■ • ■ ■- ■ .-■■■Table ■!• 

Number of Submittals that Included Data on Selected Variables 



"Variable 



SI Submittals 
n V Z 



S2 Rejections 

n %\ 



S2 Approvals 

■■ n ^ ■%■,■.. V 



Measurable objectives 
. Quasi- or experimental design 

Comparison standar(d|^: . 
. Representative sample'^ 

History threat cdntrolled 

Maturation threat^t controlled 

Testing threat controlled 

Instrumentation threat controlled 

Regression threat controlled 

Selection threat controlled . 

Mortality threat » controlled . 

Interaction wdjfh selection threat 
'^controlled 

.Ambiguity threat controlled 
Rivalry threat controlled • 
Equalization threat controlled 
Demoralization threat controlled 
Diffusion threat controlled 
Experimenter- effect threat controlled 
Hawtliorne threat* controlled 
Setting interaction threat controlled 

/S'election and treatment interaction 
controlled 

One or more instruments administered 
•Edu/cational importance (s.d. formula)^, 
(^fplication ... ^ 
^Cost information included 



7n 
/ u 


1 nn 




iUU 


1 0 • 


100 


3A 


• *♦ y 




A(f^ 

M D 


7 




6A 


91 


! 11 


85 


10 ^ 


76 


■AW* 


26 




1 

X u 




JP 


H D 


Dm 


Q 
0 


' DZ 




by 


H i. 


Dy 


' .0 


- DZ 


Q 

, 8 


oZ 




Dm 


n 

y 






69 


A3 


.61 


9 


69 


. 9 


69 • 


19 


27 


•2 


15 


A . 


31 


53 


76 / 




77 : 


' ii 


85 


AO 


57 


3 


23 




38 


8 - 


11 


1 


^ ' .. 

8 


■' ''i? • 


15 ' 


53 


76 


A 


31 


13 ' 


100 


5 


■7 






.•1 ■■ 


8 


. 2 
3 


3 
A 






• * ■ ■ ■ • 




2 


3 










17 


2A 


■■ " 3-.' 


23. 






A. 


"6 ' 










1 


1 

t 










■5A 


77 


•11 


85 - 


.11 


B5 •. 


70 


100 


13 


100 


13 


100 


35 


50 


.3 


' 23 


. ■ 7 


; 5A ■ 


A3 


,61 


7 


-5A 


9 


69 r 


37 


53, 0 


■ 7 ■■ 


; 5A 


: 7 





1. * 



Figures included those projects that used a national normative group. 

'^Figures included) only those projects t^hat used "either , probability sampling 
or a combination of probability-nohprobability procedures in obtaining the treat- 
ment group. , • 

^Figj^res included those /projects that provided information' about the inter- 
vention's educational importance for some or all test results. ' • ' 

^Figures included only those projects that provided information, on estimated 
costs for installation and operation* v 



Variable 



I 

V 



: Table 2 • ■ • ; 

Niimber of Submit tals^t hat Included Data 
on Variables Related to Standardized 'Tests 



SI Submittals 
n % 



S2 Rej ecti ons 




'Followed publisher's ^guidelines 

Reported all test^esuits 

Statistical significance of «. 
'< a • 

reported *test results 

Report i^d test statistic/^ 

Inconsistencies in reporting 

' \ " 



35 


88 


' ,0 




10^ 


25 


' 6 


67 


21 


53 ' 


■ 4 


a 


28 


70 


7 


.78 


9 


'23 


7 


78 



^Figures included -those proj ectis that provided data related to the statistical 
significance of some or all test results. 



Table 3 

Number of Submittals that Included Data 
on Variables Related to Locally-developed Tes 



Variable 



SI Submittals ' S2 Rejections S2 Appwvars 
(n « 33) (n = 8) (n - 7) 

. n % n % ' n X 



Used standardized test\ - 
administration procedures 
.Reliability of test ^reported 
Validity of test reported 
Repot t-ed all test results 
Statistical significance, of 

reported \tes.t resuults^ 
Repor^d te^t statistic 
Tnconsistencies in jrii^porting' 



13 


39 


0 




0 




23 . 


70 


5 


62 


. 5 


* 71 


19 


58 " 


2 


25 


3 


41 


28 


85, 


8 


100 


. 7 


100 


18 


55 


2 


25 


5 


n 


26 


85 : 


8 


100 


7 


■ 100 


12 


36 


2 


25 


. 2 


29 



V^'Figures "included those projects that provided data related to the statistical 
sl^if icaiice of 6pme or all ' 



In ithe 'two sub-sampies approved and rejected submittals were compared 
to obtajfn the. answer to research question 2. • Chi -square* tests slgnifi-. 
cance and t-iests were employed to determine whether there werei.Vtati>!s£ically 
cantv differences. In the sub-sample where^ projects submitted on only 
asimj results showed that approved submittals si gn.ifi cant Ty differed 



signifi 
ome dec 

from rejected submittals" in the. following ways: 



1. they* reported using a higher number. of testing measures; 
2.. information, was ^provided on the salient features of the * 

intervention; . - . • . 

3. the educational importance of the intervention was 
discussed iji the narrative; and .* . 

4. the information in the t^ibles wasvQl early presented. . 

In the sub- sample where projects. submitted on two occasions, findings iridi- 

' ■ ■■■■ ■■ ' ^ \ ■ ' ■ ' 

cated that approved submittals significantly differed from rejected sub- 

- ■ - ' ■ ' ■ ■ ■ ■ ■• ■ ■ 

mittals .in the following ways: - ** 

' ^ ■ • . ' ■ . ■ \ - --^ ■ ■ X ' ■ ■ ■ 

\ --^ 1. they repdirted controlling a higher aumber of jnternal \ 

validity threats;. . 

2. they reported using a higher* number of testing measufes; ^ : 

3. reported results were statistically significant; . 

4. infonnation was pro\nded ohrthe salient features of the 
intervention; and ' , 

:^ \ ' , ^ ■. U ^ 

' 5. the information in the narrative and tables was clearly 
■ ^ ' presented. \^ ^ 

In both sub-samples, approved submittals tended' to provide information, 

•■ ■ ■ . ■ . ■ ,■ ■ • ■ ■ • ' . ' ■.; 

related to the educational importance of the reported test results. y 



/ 



Educational importance was defined using the JDRP-^anctioned standard 
deviation formula,^ (Y^ - Y^)^ (tfV2 of the normative group). Tables 4 



and 5 present .-the stati stics on tfio^variabTes in which statistical- 



significance* was found. 



Table 4 



Statistically Significant Differences 
That Were Found Between Approved and Rejected Submittals 
That Were Reviewed on Only One Occasion 



Variable 



Scale 
Values 



Reject- 
ion X 



Approv- 
al X 



Statistic 



df 



Number of test 
measures ^ 

Salient features 

Educational import- 
ance discussed in 
narrative 

Educational 
iijiportance (s.d. 
formula) 

Clear tables ^ 



U2B 
1-3 

1-2 

1^3 



1.2 
2.2 

V ■ 

1.4 
1.6 



2.5 
2.5 

1.8 

1.8 
2.3 



t = 2.4 
t ^ 3.0 



^= 4.9 



t = 1.9 
t = 3.3 



54 
42 



68 

68" 



.0 
.0 

,0 

.0 

-.0 



- Table 5 - 

Statistically Significant Differences 
Tha^ Were Found Between Approved and Rejected"Submittal s 
That 4/er^-4^ 



1 

Variable 4 


. .Scale • 
Values 


Reject- 
ed X 


• 

- Approv- 

- al X- 


Statistic 






Number of 

controlled internal 
validity threats 


r- . 

.1-13 


_ 


.5.5 


t= 6.6 


/ 

25 


.001 


»- 

Number of -test 

measures 


1-28 


1.5 


. 1-9 ■ 

■ ^ 


"^""^ 


25 


,05 


• Statistical . 
significance 




2.1 


} . 2.8 


t= 4.5 • 


25 


•oo( 


*Salient features 


1-3 


1.9 • 


2.5 


.t= 4.9 : 


25 


' . 001 


, Educational 
importance ; » , 
(s.d., fonnuTa-) 


1-3 


1.2 ' 




; t= 2.1 


12 


.06 


Clear narrative 


' 1-5 • ' 


2.5 


3.5 


t=-5.9 


25 


.001 


Clear tables 


1-3 




2.4 


t=2.1 


25 


.05 



' • ^ ■ •■ . " . . " . ' ; • I 

To answer research question 3, a* discriminant analysis procedure was 
performed oh variables related to JDRP review criteria. Only jDrojects that 

- - ■ . 4 . ... . . ■ ■ * • . 

.subniitted_pnce to jDRP_were cojrvsidered in the analysi.s. - AnalysisVdempn- 
strated that six variables- w^re directly related to the apprbved-^sybmittals. 
These approvals reported the use of quasi -experimental or experimental de- 
signs more frequently than pre-experimental. designs, a higher number of 
tests, a greater amount of 'information on the intervention's salient 
features, clearer tables, and the elimination of selection and statistical - 
regression effects. Only one variable, the statistical significance of 
test results, was inversely related to the function. These results pror 
vided additional evidence that approvals coul-d be distinguished from 
rejections on selected variables related to the intervention's effectiveness. 
\ Table 6 presents these data. 




. -Table 6 

Variables that Account for the Function 
Using. Step-wise Discriminant Anaiyiis Procedures 

' "■ ' ■ ■ r ' . — ^ 



' -prediptor Scale' DiWi mi nanf Function Wilk's Level of 

VaViable : Values. Coefficient ' Lambda . , Significance 

Selection - „ , ^• 
threat . * 

controlled - 1-2 > / -.60^ .78 .001 

Regression > • 

threat ( . - - 

controlled 1-2 . : -.49 V^ . .. .74. r ' • -001 . 

Information * : ' 

. . • X)f salient ■ \ ' ' nm 

■features 1-3 * '^^ '^^^ 

• ■ . Evaluation ; ' 

design. •, .n-2^^^^^ ^^^^^ : V - / -^^ : . .001^ 

' Stati'stical . .■■"-'^ /.■■.^ . . . ' 

:..sfgnifTcan(:e. ■■■■ - . ■ vj ■ ' : " ■.. ' 

of reported C. .. . . . 

. .. tests - \ W /■ :, .35 : V . ..65 : -^^ ^ ...^^^^ 

' Numb6r of - " ^ " nnr ^ 

.^..:instruments_ lr28. ; 31. "'-^^ -OPl,; 



• Note . .Percent' of • explained variance was 37.. 5%. Group X for approvals was 
-.44; group X for rejections was 1.30. 



> ■;■ 
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The results of this study have implications for three types of - / 
audiences— program developers/ demonstratorl,^ potential -adopters/adapters 
at the local level , and JDRP members/ In terms of the program ^developers 
who a»^e considering submission to JDRP, the study has demonstrated that 
all submittals, regardless of the final Panel decision, should contain a 
statement of the. objectives written in measurable terms, an evaluation ^ 
design that utilizes a comparison standard and a specified sample size 
for the treatment group, and should provide evidence that at least one - 
instrument was employed to measure the intervention' s effects. The use 
of multiple measures is recommen(Jed since findings indicate that the higher 
the number of testing measures that were implemented, the greater the 
chances of approval. Results also "emphasize the need for projects to - 
. provide information on the key features of the educational practice or 
product. A clear description of the essentiaMnterventi on components 

enables local educators to make a decision about adopting a particular 
' intervention. The submittal should also contain a discussion of the 
educational importance of the practice or product. Program developers 
.are also cautioned to select evaluation designs in whkh the effects are 
clearly attributable to the: intervention". Approved submittals tended to ^ 
show evidence that the third-variable threats of selection,* history , and r 
statistical regression were controlled. Although there , was no statistical ' 
significance between approved and re j.ected submittals on the type^o'f 
V eval uation design (i,. e. i pre-experimental , quasi -experimental , experimental ) 
that was employed ♦ In interactions with panel members several -noted a 




personal bias toward the use of experimental designs. On the other handjj^ 
the Education Division *(Note 1) and other Panel irlembers have .stated that 
an experimental design is not necessary as long as data are presented' 
that show a comparison standard was utilized. 

Findings demonstrate that the type of testing measure (i.e., . 
standardized vs. locally-developed) that is employed does not affect the 
JDRP decision. However, if the submitter decides to use a locally- 
developed test, it Is advisable to report reliability and validity in- 
formation. Although no statistically significant differences wpre found 
on this variable, both JDRP submittal guidelines and several Panel members 
emphasize this point. 

No statistically significant differences were obtained when the 

• irtyes.tigator looked at the type of da.ta that were riSported for each'of . 
the tfesting measures, in the. submittals.. However, when one reviews the 
dat? related to implementation, analysis i and repoftlTi^variables (see - 

• tables ii2,2ind 3) it is evident that projects in CTe suW-^ample that . 
.submitted on two occasions included more data in their resubmittals. 

This 'is particularly true in terms of the test results; the test statistic; 
and informatipn on replication. . 
S - In terms of the local practitioners who are considering the adoption 
of a JDRP-ajDprbved educational practice or produ^, they will generally 
find that information related to measurable objectives,, the use of a 
comparison standard, the reporting of. rel iable and val id measures, and the 
■ TtatisticaY signific|nce^o^ reported tests is. usually available; . 
Approximately 501 of the time, the intervention will have data related to 
the educational importance of th,(5 practice or product (i.e.,;in terms of 
the standard deviation formula); replication figures for more Aan one site' 



or time period; and cost effectiveness infolrtnation. This cost effective- 
hess information -may not give an accirrate portrayal of the installation 
and recurring costs of the intervention. Accuracy will depend on the 
developer's reiporting of the individual jco^ponents t^^^ comprise each 

^estimate. . ' 1 

Information concerning the control of internal and. external validity 
threats will vary from submittal to submittal. Although selection effects 
and the ambiguity of the causal direction were frequently addressed 
X83%.and8U respectively), the elimination qf other "third-variable 
threats was not as high (see Table 1). The contrbl of external validity 
threats was almost completely lacking. This is -distressing when ^viewed^ ^ ^ .^^^ 
from the perspective that these approvals are eligible for national dis/ 



^ ... 

semi nation^ and adoption. 



■ Panel' members may profit from this study. in terms Of seeing how ah 
external evaluator has systematically examined a sample of the reviewed 
submittals. ^ In interviews Panel members emphasized the importance of 
nieeting each of the establi-Shed. review, criteria , ;yet tW data cl early 
:. demonstrate , that oall approvals do^ not meet eadh of the^e. standards; ' 

This is particularly the case in the criteria of attribution (i .ei , the ;, 
' eliminiation of alternative explanations) and generalizability (i.e. , the 
V ' control of external validity threats) . Of those variables that different- 
iated between the approvals and rejections, three were su6j6ctive measures — 
the clarity of the tables, the amount of information presented on all salient 
features of the intervention, and the' discussion of educational importance 
in the narrative. Perhaps additional variables are operating that would 
provide further cliiriffcJiti on of the revi^Sl process. ' . 



V 



Findings demonstrated that/ there Were a f€ 



single variables which ' 
On the other hand, if 



iistinguished between, approyals and rejection 

■.one looked at a cluster /of var/iablesJ it was I'ouhd that a combination 
Of seven variables discri mi nated beti^een the/approved' and rejected * 
• submi this. These findings liighlight the n|'ed for further studies in 
this area. A replication of the present stydy shouW be conducted, 
whereby the review criteria; are verified across time and tha use of the 



mihites from. the rev-iiew sespi-ons is a part of the research procedures. . 

Other areas of future study arq thje exploration of the feasibility of 
' developing more detailed guidelines fpr potential submitters (e.g., 

presentation of cost effectiveness information, reporting of replication 
• data) arid the examtn^tion of the role of 'additional factors (e.g^ 

Panel variable^. Education Departnieijt"' reorganization) that may haye a 

bearing on the review process.-^ ' " ' ; , ' 
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